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ABSTRACT 



This paper considers the feasibility of linking a student 
data sample with the Schools and Staffing Survey (SASS) teacher and 
administrative data. The National Center for Education Statistics (NCES) has 
from time to time considered linking their student-based elementary and 
secondary surveys to the school- and teacher-based surveys. Speculation on 
the feasibility of linking datasets is of particular importance in the 
current climate of budgetary constraints and distrust of federal data 
collection. The rationale for linking a student data collection in SASS is 
discussed, and some options for collecting these data are explored, whether 
linking the National Assessment of Educational Progress to the SASS or 
linking a new National Education Longitudinal Study with SASS. The argument 
for attaching a longitudinal component to the SASS response rests on several 
premises . Doing this could measure the types of student data deemed most 
worthwhile, while satisfying the two criteria for sensible merger: producing 
some cost benefit and engendering an analytical payoff. Of the various 
options available, merging a new National Education Longitudinal Study and 
the SASS provides the most benefit to the whole educational policy community. 
(Contains six references.) (SLD) 



******************************************************************************** 



Reproductions supplied by EDRS are the best that can be made 
from the original document . 



******************************************************************************** 



o 

ERIC 



ED 418 984 











‘ . '-• .j .• ' 



Linking Student Data to SASS: Why, When, How 





Working Paper No. 96-23 



October 1996 









*5. 







'- • ’ ■ ' * y-f- 






{ T 









. A V ■ rf- 










{3^^^ 
















rrw 













, • -■• « 



' .*X -'-^V -J-- 




education 

Office of Educationaf Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

cj This documGnt has bean reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. ^ 



o 

ERIC 







Linking Student Data to SASS: Why, When, How 



Working Paper No. 96-23 



October 1996 



Contact: Dan Kasprzyk 

Surveys and Cooperative Systems Group 
(202) 219-1588 

e-mail: daniel_kasprzyk@ed.gov 



U.S. Department of Education 

Richard W. Riley 

Secretary 

Office of Educational Research and Improvement 
Sharon P. Robinson 
Assistant Secretary 

National Center for Education Statistics 
Pascal D. Forgione, Jr. 

Commissioner 

Surveys and Cooperative Systems Group 
Paul D. Planchon 
Associate Commissioner 



The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, 
and reporting data related to education in the United States and other nations. It fulfills a congressional 
mandate to collect, collate, analyze, and report full and complete statistics on the condition of education 
in the United States; conduct and publish reports and specialized analyses of the meaning and significance 
of such statistics; assist state and local education agencies in improving their statistical systems; and review 
and report on education activities in foreign countries. 

NCES activities are designed to address high priority education data needs; provide consistent, reliable, 
complete, and accurate indicators of education status and trends; and report timely, useful, and high quality 
data to the U.S. Department of Education, the Congress, the states, other education policymakers, 
practitioners, data users, and the general public. 

We strive to make our products available in a variety of formats and in language that is appropriate to a 
variety of audiences. You, as our customer, are the best judge of our success in communicating 
information effectively. If you have any comments or suggestions about this or any other NCES product 
or report, we would like to hear from you. Please direct your comments to: 

National Center for Education Statistics 
Office of Educational Research and Improvement 
U.S. Department of Education 
555 New Jersey Avenue, NW 
Washington, DC 20208 



Suggested Citation 

U.S. Department of Education. National Center for Education Statistics. Linking Student Data to SASS: IVhy, When, How. 
Working Paper No. 96-23, by Phillip Kaufman. Project Officer, Daniel Kaspryzk. Washington, D.C.: 1996. 



October 1996 



Foreword 



Each year a large number of written documents are generated by NCES staff and 
individuals commissioned by NCES which provide preliminary analyses of survey results and 
address technical, methodological, and evaluation issues. Even though they are not formally 
published, these documents reflect a tremendous amount of unique expertise, knowledge, and 
experience. 

The Working Paper Series was created in order to preserve the information contained 
in these documents and to promote the sharing of valuable work experience and knowledge. 
However, these documents were prepared under different formats and did not undergo vigorous 
NCES publication review and editing prior to their inclusion in the series. Consequently, we 
encourage users of the series to consult the individual authors for citations. 

To receive information about submitting manuscripts or obtaining copies of the series, 
please contact Ruth R. Harris at (202) 219-1831 or U.S. Department of Education, Office of 
Educational Research and Improvement, National Center for Education Statistics, 555 New 
Jersey Ave., N.W., Room 400, Washington, D.C. 20208-5654. 



Susan Ahmed 

Chief Mathematical Statistician 
Statistical Standards and 
Services Group 



Samuel S. Peng 
Director 

Methodology, Training, and Customer 
Service Program 



Linking Student Data to SASS: Why, When, How 



Phillip Kaufman 
MPR Associates, Inc. 



October 1996 



Table of Contents 



Page 

Foreword iii 

Some Ground Rules 1 

Why Collect Individual Student Data through SASS? 3 

Linking NAEP to SASS 5 

Linking a New NELS with SASS 7 

The Design of the National Longitudinal Study of 1998 10 

Periodicity of SASS 11 

Summary 11 

Conclusion 12 

References 13 



Endnotes 



14 



LINKING STUDENT DATA TO SASS: WHY, WHEN, HOW 

Phillip Kaufman, MPR Associates, Inc. 



This paper considers the feasibility of linking a student data sample with the SASS teacher 
and administrative data. NCES has from time to time considered linking their student-based 
elementary and secondary surveys to the school- and teacher-based surveys. These thoughts 
have usually been centered on the analytical power that such a student/teacher data set would 
hold. Budgetary concerns— in terms of both fiscal program budgets and burden 
budgets — have also been important, but the analytical justification of linking student data to 
teacher data has generally taken precedence. However, speculation on the feasibility of 
linking datasets is of particular importance now in the current climate of budgetary 
constraints and distrust of federal data collection among segments of the public. 

NCES (and indeed all of the governmental statistics community) is entering an era when hard 
choices need to be made concerning data collection and reporting. In an era when we face 
increasing demands for more and better data from a wide variety of educational policymakers 
and researchers, we are also being asked to do more with fewer resources in terms of both 
program money and personnel. Thus it may no longer be feasible to collect data on schools, 
teachers, and programs through the Common Core of Data and the Schools and Staffing 
Survey, conduct another National Assessment, continue to track the early childhood cohort of 
students (in ECLS), and launch a new longitudinal study of high school students. If it is not 
feasible to collect all of these data as they have been collected in the past (regardless of their 
analytical merit), it seems to me that ways must be found to collect all of these data with 
new methods or systems. Unless we think hard about these issues, opportunities may be lost 
and we will create gaps in our knowledge about American schools and the impact of the last 
few decades of reform. Old ways of conducting the business of data collection may have to 
be modified in light of the increased budgetary constraints imposed by Congress and the 
President and the simultaneous increase in analytical demands of the public. 



Some Ground Rules 

In this paper I discuss the rationale for linking a student data collection in SASS and then 
explore several options for collecting these data. However, before launching into the main 
body of the paper, I would like to lay out a few principles to organize my discussion of these 
issues. These guidelines have to do with (1) what dimensions of student data should be 
collected, and (2) what criteria should be used to judge the merit of the proposed new data 
collection system. 

Dimensions of student data. As I will try and develop further in this paper, it seems to me 
that at least three aspects of students are important to track and should be a part of any 
system of student data. While perhaps not necessary elements of a SASS student component. 



these elements should be (and are) part of the overall data strategy for NCES and should be 
considered when thinking about what kind of student data should be attached to SASS. 

These elements are: overall academic performance, growth in achievement, and successful 
transition into and through the increasing demands of schooling and work. 

Among these three, measuring overall academic achievement is perhaps most important (for 
why else can we justify public and private investment in schools) and has traditionally had 
the most attention. Both NAEP and the longitudinal studies have made estimates of overall 
achievement levels of various groups of students over time. Accurately measuring growth in 
achievement (especially in observational/survey data) is perhaps the most challenging. This 
has been done by analysts using both HS&B and NELS;88. Examining critical transitions 
has historically received the least attention, but has come under increasing scrutiny as the 
educational community has realized the importance of studying the life-course and its impact 
on education (Pallas, 1993, p. 20). It seems to me that three main transition periods are 
important to keep in mind while considering student data within SASS: 1) the passage from 
middle school to high school, 2) the path through high school to graduation, and 3) the 
transition from graduation to school or work. 

Evaluative criteria. While keeping these three elements of student data in mind, I need to set 
up a few criteria to judge the worthiness of any proposal to linking individual data with 
SASS. It seems to me that for such a merger to make sense, it must pass at least one of two 
tests. First, it must make sense in terms of efficiencies of administration or respondent 
burden. That is, it must prove to save budgetary resources (either program budget or burden 
budget). Secondly, it should make analytical sense. That is, the merger should result in a 
data system that allows more comprehensive and sensible inferences to be drawn. 

To summarize, I start this discussion considering three main elements of student data and two 
principles of an adequate argument for linking student data within SASS: 

Elements of a student data system: 

• overall academic performance, 

• growth in achievement, 

• and successful transition through the increasing demands of schooling and work. 
Criteria for judging the adequacy of including a student component in SASS: 

• The merger should produce some cost benefit, and 

• The merger should engender an analytical payoff. 

All of the above must also be considered in the context of the mission of NCES (as I 
paraphrase it): 1) providing to the public accurate information on the “Condition of 
Education,” 2) producing policy-relevant but policy-neutral research reports on current 
and/or enduring issues in educational policy, and 3) providing databases that other analysts 



can use as research tools in their own policy work. The data needs differ for these three 
functions— ranging from fairly descriptive data for function 1 to data for function 3 with the 
potential for multivariate analysis and “cautious causal” analysis. 



Why Collect Individual Student Data through SASS? 

Nluch of the data that NCES collects are not on individual students, but are on characteristics 
of schools and other educational institutions. For example, SASS now collects data on 
school districts, schools, principals, and individual teachers. Detailed information is 
available on characteristics of the curriculum, qualifications of teachers, school and district 
level policies and practices. Traditionally, what student data have existed in SASS were 
generally aggregated to the school level before being captured. For example, percent of 
students receiving free lunch, percent of students of various racial-ethnic groups, etc., have 
been attached to the school files since the first cycle of SASS. 

However, while it is important to be able to accurately measure and track schools, teachers, 
and curriculum practices, these data provide the context for measuring the main component 
of education student achievement, growth, and progress. As the nation tries to assess and 
track the implementation of school reform, the data on schools and teachers do provide 
valuable indicators of the extent of reform— and these data have been used effectively over 
the last decade for this purpose. However, these data are much more policy relevant when 
used in the context of seeing how they are related to individual student achievement, growth, 
and experiences. 

It is possible to attach to SASS additional aggregate measures of student characteristics and 
outcomes. In fact, this is the approach advocated by Don McLaughlin in his response to an 
earlier version of this paper. McLaughlin makes the case for aggregate data based on the 
tremendous improvements in the assessment practices of many state departments of 
education. He advocates using these state assessment data (presumably available for each 
school in SASS) by linking them to the state NAEP assessment. Dave Thissen has conducted 
such equating for the North Carolina state assessment. 

I appreciate McLaughlin’s contribution to this discussion and his comments on using state 
assessments are well reasoned. However, as he acknowledges, cross-sectional data on 
student outcomes is less interesting than longitudinal data (and, in my opinion may not be 
worth the effort of collecting at all). Collecting longitudinal aggregate data on student 
achievement within schools is of more interest, but (again in my opinion) not as useful as 
collecting individual student data. That is, aggregate test scores or mean outcomes do not 
capture the individual variation in achievement that traditionally has been of such interest. 

For example, the variance of test scores within schools has been used as an outcome measure 
in assessing the effectiveness of schools. High mean test scores may be due to the school’s 
efforts at increasing the learning of students already achieving at a high level or may be due 



to the school’s successful attempts at raising the scores of students at the bottom of the 
ability ladder. Mean scores mask these important differences in the impact of school policies 
and practices. Furthermore, the transitional experiences and out-of-school experiences are so 
important to those educational outcomes. Therefore, while it may be worthwhile for SASS 
to explore the possibility of attaching aggregate and longitudinal assessment data to their 
regular data collection, it seems to me that still accurate estimates of the associations of 
teacher and school characteristics on student outcomes necessitates the linking in some way 
of individual students (or similar groups of students) with individual schools, teachers, or 
policies (or similar groups thereof). 

Of course, an expansion of the current SASS student survey (based on administrative records 
of students of sampled teachers) could add immeasurably to the analytical power of SASS. 
This option would build on the current efforts to include student data in SASS. While 
current student samples would have to be increased to be representative of the school, it still 
seems reasonable that this would be the most cost effective choice. However, it is perhaps 
the least effective analytically. Only limited kinds of data could be collected by 
administrative records— race-ethnicity, sex, absences, maybe grades. Test score data that 
would be comparable across schools would not be available. Furthermore, while data on 
dropout status may be available from administrative records, we have known for a long time 
that these data are unreliable as indicators of student status. They may be reliable indicators 
of what that school thinks is the status of the student, but that student may have enrolled in 
another school (perhaps an alternative school) or may have taken the GED and received an 
alternative credential. Student data would also be cross-sectional and vulnerable to all of the 
weaknesses of cross-sectional data. 

Thus, attaching only individual administrative student statistics to school and teacher data 
would miss invaluable insights that are derived from observing student outcomes and 
transitions in the context of student’s prior experiences, aptitudes, and ability levels in 
school— data that can only be measured through individual student surveys. Administrative 
data also would fail to capture or measure the impact of the transitions that students make 
through different schools and classrooms to the world of work and family life. Clearly, 
while collecting student data through administrative records may be cost effective, they do 
not provide the kind of data that add as much to the analytical power of SASS— only 
individual student data can do this. 

Over the years, NCES has relied primarily on two vehicles for collecting data on individual 
students— the National Assessment of Educational Progress (NAEP) and the system of 
longitudinal studies including the National Longitudinal Study of 1972 (NLS:72), the High 
School and Beyond study (HS&B), and the National Education Longitudinal Study of 1988 
(NELS:88). As an integral part of these data collections, individual student data have been 
directly linked to data about the student’s teacher, classroom, and school. NAEP and the 
longitudinal studies accomplish this by including school and teacher questionnaires along with 
student background and assessment data. Data on student outcomes can therefore be linked 
with data on educational context. 



However, much of the school, teacher, and classroom data collected by the student-based 
surveys are collected in more breadth and depth in SASS, or in any case is redundant with 
data collected by SASS. Furthermore, SASS collects data about schools, teachers, and, most 
importantly, school districts that are not collected by NAEP or the typical longitudinal study. 
In a time of tight budgets (that may become even leaner) a reasonable question is why not 
borrow the strengths of both types of surveys and link the more detailed student data NAEP 
or a NELS to the richer teacher, school, and district level data in SASS? In this manner each 
may provide contextual data to better interpret the other and possibly reduce the overall 
respondent burden (although perhaps increasing the burden on those sampled)— thus fulfilling 
the requirement I set for myself in the introductory section of this paper. This is the topic to 
which I will turn next. 



Linking NAEP to SASS 

Advantages of a Linkage with SASS 

NAEP has several distinct advantages over a NELS in such a linkage. The primary 
advantage is in the content detail that is provided in the assessment and the age or grade 
coverage available in NAEP. Due to an adaptation of matrix sampling called balanced 
incomplete block (BIB) spiraling, the design of NAEP allows for broad coverage of 
curriculum content while minimizing the burden to individual students. For example, while 
no student takes all test items, the 1992 NAEP mathematics assessment contained 178 items 
at grade 4, 205 items at grade 8, and 201 items at grade 12. This allows reliable estimates 
across 5 content areas in mathematics as well as 3 ability areas. (The mathematics 
assessment in NELS:88 in contrast, contains only 40 items and 5 proficiency levels.) 

NAEP also includes a student questionnaire that solicits background information on each 
student. NAEP is built to obtain good estimates of proficiencies in a variety of areas for 
groups of students. One of the primary strengths of NAEP is its ability to track the overall 
achievement levels of U.S. students over decades of time. From the early 1970s NAEP has 
reported on the mathematics and reading achievement of elementary, middle school, and high 
school students. This has provided educational policymakers and the general public with an 
immeasurably valuable tool in monitoring the health of our educational system. 

Weaknesses of a Linkage with SASS 

While NAEP has some obvious strengths as a candidate for merger with SASS it also has 
several weaknesses. Those aspects of NAEP that do not lend themselves to a merger with 
SASS are analytical more than procedural. For example, the main weakness of NAEP is that 
it is not longitudinal. Merging a cross-sectional SASS and a cross-sectional NAEP would 
still result in a cross-sectional survey. While the cross-sectional design of NAEP allows for 
rich data for descriptive indicator work, the merged data set with its rich contextual data and 
assessment data would still be of little use in producing valid analysis of the association of 



school policies and practices. In fact, the existence of such a dataset may actually encourage 
“invalid but potentially influential studies of schools effects that could seriously distort 
policy.”* That is, secondary analysts (or, with due apologies to William Raspberry, a 
columnist looking at published NAEP reports) could make erroneous conclusions about 
school policy based on the real but misleading associations in the data. 

Another analytical weakness of NAEP is that it does not contain good measures of student 
socioeconomic status^ (and may never contain such measures). Without a measure of this 
kind, it is difficult to accurately describe the contribution of school process and policy 
variables on student outcomes. Most of these process variables are related to student 
socioeconomic status and/or student body socioeconomic status. Again, invalid but 
persuasive inferences could be drawn from these data. 

However, while socioeconomic status is a prominent gap in the student background variables 
provided by NAEP, it is only one of several variables that one would want to collect and 
measure in order to make satisfactory inferences from associations found in the data between 
achievement levels and school characteristics and practices. These variables include, but are 
not limited to self-concept, attitudes toward school, and peer group attitudes and opinions. 

As mentioned above, one of the major contributions of NAEP is the trend data that it 
provides on student achievement in the United States. This strength of NAEP however, 
proves to be one of the greatest arguments for not linking it to SASS. It seems unreasonable 
to expect that such a linkage could be done without some modification of the design of 
NAEP— either in its sampling design or its administration design. Such changes in the 
design of NAEP could result in changes in the estimated proficiency levels in the United 
States.^ In addition, SASS is a fairly new and dynamic dataset. Again, given the 
importance of the NAEP time series, one would want to be very cautious in any changes to 
the design of SASS which would effect the design of NAEP, in either content or sampling 
design. Therefore, locking the design (and administration) of SASS to NAEP would make 
future changes in SASS very difficult. For example, currently NCES data collections poorly 
measure the classroom experiences of students. That is, while being able to describe 
educational inputs— students, teachers, schools— they do not measure educational processes 
well— what actually goes on inside the classroom. There would be many issues in 
incorporating a sample of classrooms within the design of SASS— including preserving the 
trend data of schools and teachers from earlier rounds of SASS. Adding the encumbrance of 
insuring that the trend data from NAEP is also preserved would make this task even more 
difficult. 

Furthermore, while merging the two surveys could produce savings in total respondent 
burden to the educational system, it almost certainly would increase respondent burden for 
individual schools and teachers that are sampled in the merged survey system. This could 
result in lower response rates and threaten the data quality for both surveys. NAEP has 
traditionally relied on high response rates to insure the quality of the trend data. Again, in 
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my opinion the integrity of these data is too important to jeopardize in a SASS/NAEP 
merger. 

The NAEP emphasizes the production of reliable estimates of national and state achievement 
levels. Consequently, NAEP does a good, but not perfect, job of estimating the first element 
of student data I outlined above— measuring overall student achievement. However, the 
strength of NAEP is in measuring aggregate-level measures of proficiency and not individual 
I or school-level measures of proficiency. The capture of individual proficiencies or 

achievement levels has never been the main goal of NAEP. Given the complex nature of the 
plausible value methodology, individual or small group proficiencies are measured with a 
good deal of measurement error. 

^ NAEP is also a survey that emphasizes content depth over breadth of background variables. 

The burden budget of NAEP goes into accurately measuring content. Student background 
coverage is not ignored, but certainly has less emphasis than in the longitudinal studies. The 
longitudinal studies, on the other hand, have had somewhat different goals. For example, 
while NELS:88 also aspired to provide accurate estimates of group proficiencies, it had the 
added burden of obtaining accurate estimates of school and individual level proficiencies and 
® individual growth. There was also the emphasis in NELS:88 on the measurement of a 

variety of student educational outcomes and not just academic achievement. To control 
respondent burden, the academic assessment tools in NELS:88 had to be much shorter in 
scope and content than the NAEP assessment. NAEP puts its burden dollars in the depth of 
the content while NELS:88 put its burden dollars in breadth of outcomes and background 
• information. 

Furthermore, because NAEP does not measure students longitudinally, it does not do a good 
job of measuring (and does not attempt to measure) the other two elements of my list of 
student data above— growth in achievement, and successful transitions through the increasing 
P demands of schooling and work. Lx)ngitudinal studies are needed to track these types of 

outcomes. For these reasons a new NELS (or some modification of NELS) may be a better 
candidate for merger with SASS. It is to this topic that I turn next. 

i Linking a New NELS with SASS 

While it is important to measure and track overall achievement levels, it is also important to 
be able to associate differences in school policies and practices with student achievement. It 
is almost impossible to make valid inferences about the impact of school policies with cross- 
^ sectional data— regardless of how rich the individual data may be. Of course, making clear 

inferences about these kinds of associations is done best by experiments in which students are 
assigned to educational treatment conditions and subsequent growth in achievement is 
measured (Metcalf, 1995). 
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However, true experiments in education are difficult to conduct and maintain under the best 
of circumstances. Many educational researchers have therefore relied on observational 
survey data to make cautious inferences about policy effects on achievement gains. While 
these studies have many well known inherent flaws, most educational researchers and policy 
makers have been determined to not let the “perfect be the enemy of the good” and have 
conducted well thought out and executed policy smdies with the longimdinal smdies data 
systems provided by NCES (Heyns &. Hilton, 1982, pp. 89-102). 

Three Options to Consider 

It seems to me that there are at least three options to attaching a longitudinal smdent 
component to SASS. These are outlined below. 

(1) Attach student administrative data to SASS and return to those schools to pick up 
longitudinal data. This option would be substantially more expensive than simply 
attaching smdent administrative record data to SASS since one would have to remrn 
to the SASS schools to follow up on the smdents sampled in the first year. SASS is 
currently on a 5-year cycle. Presumably one would want to go back to recapmre 
smdent data on a more frequent follow-up schedule— perhaps every 2 years. Re- 
surveying schools every 5 years to follow up on smdents is perhaps too long a 
periodicity to make timely estimates of smdent outcomes. One could of course go 
back to the SASS schools (or sample of SASS schools) to capmre just those 
administrative records that one needs. However, even this would increase the 
administrative and respondent burden of the survey system without providing much in 
the way of analytical payoff. Smdent test data would still not be available and 
consequently measures of growth in achievement would also not be available. In 
terms of measuring transitions, one would know if smdents were still enrolled in that 
school, but would know precious little else about the smdents transitions to other 
school or work. Furthermore, some portion of the smdents would have moved, 
making followup of their stams difficult and expensive. 

In addition, learning takes place in an interaction of school, home, and family. A 
smdent data collection based solely on school records obviously records only one 
aspect of this learning system. The longimdinal smdies have long recognized this and 
have tried to measure the other aspects of the smdent’s learning environment. 
Measuring only one component does not allow one to fully examine the totality of the 
smdents’ learning experience and how the different components interact with one 
another. 

(2) Create a new longitudinal survey and “link” several items to SASS items. NCES could 
field a new NELS with either an eighth-grade or tenth-grade cohort and use identical 
items from SASS in its school and teacher questionnaires. Linking these data would 
provide some analytical payoff in terms of generalizabilty of the data provided. It 
would also decrease the burden to individual sampled schools, which would 



presumably not have to respond to the both the SASS and NELS survey instruments. 
However, it would increase overall response burden and would likely increase overall 
administrative costs. The analytical payoff would also be somewhat weak, since the 
linked data to SASS would not include all of the contextual data provided by the new 
NELS. 

(3) Merge a new NELS with SASS. NCES could field a new NELS in a sample of SASS 
schools. For example the 1998 SASS could become the base year of NELS: 98. The 
overall analytical reward of such a merger could be substantial. This class of students 
will be on schedule to graduate in 2002, thus leading to clean comparisons among the 
high school classes of 1972 (NLS-72), 1982 (HS&B), and 1992 (NELS:88). The 
longitudinal studies have traditionally have had teacher and school data, but have not 
have had district level data to attach to student data. Furthermore, the richness of the 
SASS teacher and administrator data would enhance the student and parent data from 
NELS. Student assessment data (perhaps both cognitive and affective) could be 
attached to the SASS data to enable analyses of the association of outcomes data with 
school and district policy information. Data would also be collected with several 
followups and would thus be able to measure growth in outcomes. Information would 
also be available to track the success of students in making critical transitions through 
school and work— for example, transitions from middle school to high school, through 
high school to high school completion, and from high school completion to 
postsecondary education and/or the world of work. 

While a new NELS attached to SASS makes sense analytically, it also makes a great deal of 
sense in terms of cost savings. The SASS data collector will have already contacted the 
schools and collected data from districts, schools, and teachers. A new NELS would only 
have to supplement these data with a student and parent questionnaire— the teacher and 
school data would be collected within the normal SASS administration. Using the 1998 
SASS survey as the base year of a new NELS has been shown to indicate a substantial cost 
savings over a separate sample design (J. Owings, internal memo, 1995, National Center for 
Education Statistics). 

While total response burden would presumably be decreased by a NELS/SASS merger, the 
burden to individual schools will almost surely increase. However, this increase in response 
burden would have the potential to effect the response rates of the NELS data collection 
effort rather than SASS. SASS should not have to pay any part of the response rate price 
associated with the merger. 

Thus, a new NELS attached to SASS would meet the requirement that I set forth in the 
introduction to this paper. It would collect all three types of data that I think is 
important— overall achievement data, data on cognitive and affective growth, and data on 
critical transitions. It would also meet the two criteria for a reasonable merger — it would 
make sense analytically, and it would make sense economically. 



However, a new NELS attached to SASS would still have to overcome several obstacles and 
several issues will need to be addressed in designing a new NELS. In fact, fleshing out a 
design for a new longitudinal study attached to SASS deserves its own design conference. 
However, short of this, I briefly outline two areas of concern in the next section. 



The Design of the National Longitudinal Study of 1998 

What age cohort should NELS:98 begin with? To track the transitions I outlined above, 

NELS: 98 could start with either an 8th-grade cohort (to follow the transition from middle 
school to high school and allow trend comparisons with NELS: 88), with a lOth-grade cohort 
(to follow the transition from high school to graduation and allow trend comparisons with 
HS&B and NELS: 88), 12th-grade cohort (to track the transition from high school to 
postsecondary education or work), or some combination of the above. 

Starting with another eighth-grade cohort has a lot of analytical appeal. The transition from 
eighth grade to high school is a significant passage. Meaningful research has been done with 
the NELS:88 cohort on this issue. Furthermore, data from NAEP and from NELS:88 
indicate that a significant amount of cognitive and academic growth occurs during this 
period. Larger gains are realized, on average, between the 8th and 10th grades than between 
the 10th and 12th grades (Crouse & Ralph, 1996). 

However, despite the intuitive appeal of starting with an 8th-grade cohort, for a variety of 
reasons a lOth-grade cohort may be more feasible at this time. The primary reason for this 
is the ease with which lOth-grade students can be followed and therefore the lower cost 
involved. While younger cohorts are perhaps always more desirable analytically than older 
cohorts, following younger cohorts is always more expensive than following older cohorts. 
For example, almost 90 percent of NELS: 88 8th graders changed schools between the 8th 
and 10th grades, while less than 20 percent of NELS:88 10th graders changed schools 
between the 10th and 12th grades. Tracking students from the 8th to the 10th grade proved 
to be much more expensive than originally estimated with the NELS: 88 first follow-up study. 

Furthermore, while there was great analytical payoff to estimating the growth in achievement 
of an 8th grade in NELS: 88, the complexities of the psychometrics involved in this effort 
were severe. Because the NELS: 88 test battery was used to measure overall achievement 
levels and growth between the 8th and 12th grades, floor and ceiling effects were much more 
worrisome that in HS&B, where growth was measured between the 10th and 12th grades 
only. The resulting adaptive nature of the NELS: 88 assessment created analytical problems 
with researchers not sophisticated with psychometrics. For example, measuring gains in 
mathematics proficiency was much more complicated than merely looking at IRT gains 
scores, as had been done in HS&B. Since different kids took different tests, gains had to 
examined in terms of gains in proficiency functioning rather than raw or IRT estimated 
gains. Again, this complication was due to the fact that the assessment instruments had to 



have a multilevel design to guard against the floor and ceiling effects that could occur when 
testing spanned the 8th through 12th grades. 

It is also interesting to speculate whether a 12th-grade cohort (either selected on their own or 
an “aged” lOth-grade cohort) could be attached to SASS in the high school years and then 
attached or merged in a new Beginning Postsecondary Student (BPS) survey when the year 
after they are scheduled to leave high school. I realize that the sampling issues here may be 
enormously complicated and can only speculate about the complexities of such an 
overlapping or multiple frame design. However, by designing the three surveys in this 
manner, one would have the merged power (and savings?) of a SASS, a NELS, and a BPS. 



Periodicity of SASS 

To parallel the structure of the HS&B and NELS: 88, the new longitudinal study should be on 
a 2-year cycle. That is, if NCES starts with a lOth-grade sample, they would want to go 
back and re-interview the sampled students 2 years later when most of them will be in the 
12th grade. In this manner, trend analyses could be run with the HS&B and the NELS; 88 
10th- to 12th-grade cohorts. Since SASS is currently on a 5-year cycle, the 2-year followup 
would have to be done separately from the normal SASS cycle. These independent follow-up 
interviews could be done either as a CATI or as in-school interviews. In-school interviews 
would probably be more costly, but would be more efficient if cognitive assessments were 
conducted during this followup. (Unless someone develops a way to efficiently do a 
NELS: 88 comparable assessment through CATI.) 

Furthermore, in many ways HS&B and NELS: 88 were multiple-cross-sectional data sets. 

Data were collected on the same people for two years apart. What went on in between those 
two data points is often hard to determine. For example, detailed information on school 
enrollment has been difficult to obtain from HS&B and NELS: 88. One knows from the 
various followups if sampled members were attending school at the time of the followup, but 
do not know much about their enrollment status in between the follow-up survey dates. One 
could use CATI to efficiently go back to these students more frequently than a 2-year cycle 
and collect such time-sensitive data. These intermediate interviews would be limited to just a 
few items (dropout status, pregnancy status, employment status) with fewer time dependent 
variables reserved for the more in-depth 2-year follow-up survey. 



Summary 

The argument for attaching a longitudinal component to SASS rests on several premises. 
First, attaching a longitudinal study to SASS seems to satisfy most of the criteria I have set 
out for myself. It could measure all three of the types of student data deemed most 
worthwhile, while also satisfying the two criteria for sensible merger— producing some cost 
benefit, and engendering an analytical payoff. The payoff, however, is to the overall data 



collecUon effort of NCES and not necessarily to SASS data collection in particular. In fact, 
attaching a longitudinal study to SASS may have no payoff whatsoever for SASS but may 
indeed provide more burden to the already overworked SASS staff. Attaching aggregate 
longitudinal student data to SASS may be of more benefit to SASS itself— merging a new 
NELS and SASS provides the most benefit to NCES and indeed, to the whole educational 
policy community. 



Conclusion 



The years 1983-84 saw the release of two publications that would forever change the way 
that Americans looked at their elementary and secondary schools. Ernest Boyer’s High 
School. A Report on Secondary Education in America. 1983 focused public attention on 
American high schools, a “troubled institution” with a confused mission and low standards. 
At about the same time the U.S. Department of Education released A Nation at Risk, which 
called attention to what was termed a “rising tide of mediocrity” in American schools. Due 
in part to the publicity these reports engendered, a decade of educational reform took hold in 
the American educational system. This “reform” was actually many reforms and debate over 
the consequences of these reforms continues today. NCES data help frame and focus this 
debate. 

In 1984, a cohort of students had just graduated (in 1982) from high school. Their 
experiences in the pre-reform era would serve as a base line to judge the impact of the 
coming reforms. The High School and Beyond study would record the experiences of this 
cohort of students. In 1984, another cohort of students was in the fourth grade. These 
students would feel some of the immediate consequences of these reforms. Their experiences 
in high school, in postsecondary education, and in the transition to the world of work were 
captured in the experiences of the students in the National Educational Lx)ngitudinal Study of 
1988. In 1984 (the year in which A Nation At Risk made its first impact), yet another cohort 
of children were bom who are right now experiencing the full impact of the reforms of the 
last two decades. Most of this cohort are on track to graduate from high school in 2002. 

Unfortunately, current budget concerns cast doubt on whether NCES will be able to field an 
independent longitudinal study of this class of high school students. The cohort of students 
who will be included in the Early Childhood Longitudinal Study will not be graduating from 
high school until 2012. Missing the class of 2002 will result in a data gap of almost 20 
years and will weaken our ability to measure the impact of the changes introduced into our 
elementary and secondary schools. Failing to capture the experiences of the high school 
class bom at the very beginning of reform will be a serious gap in the nation’s knowledge 
about education. Linking a new longitudinal study with SASS may be the only way of 
effectively filling this data gap. 
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Endnotes 



1. This quote is attributed to Dan Koretz (and similar sentiments attributed to Richard 
Mumane and Marshall Smith) in L. Burstein and P. Aschbacher (1987). 

2. A weak measure of SES has been used in several analyses using the NAEP data. See 
C. Arnold and P. Kaufman for an example. 

3. This happened in the mid-1980s with the NAEP reading assessment, resulting in the 
so called "reading anomaly." 
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